feat: Phase 10: Synthetic workload generation & telemetry validation suite - #6519
pratikmankawde wants to merge 470 commits into
Conversation
dc53faa to
d3d2325
Compare
7fb4b09 to
0a44290
Compare
3faf309 to
ea60b46
Compare
252aceb to
42a46a3
Compare
42a46a3 to
0314b25
Compare
0314b25 to
fb71866
Compare
pratikmankawde
left a comment
There was a problem hiding this comment.
CI Build Performance Issues
The Telemetry Validation workflow is timing out because:
- Conan packages are being built from source (~45 min) instead of pulled as pre-built binaries from the remote
- No ccache for the xrpld compilation step (~45+ min cold build)
Combined, these exceed the 90-minute timeout. See inline comments for specific fixes.
pratikmankawde
left a comment
There was a problem hiding this comment.
Build Performance Fixes
4 suggestions to fix the timeout. Apply all of them together:
- Add ccache env vars
- Install ccache package
- Authenticate to Conan remote (this is why deps build from source for 45 min)
- Use ccache in CMake configure
pratikmankawde
left a comment
There was a problem hiding this comment.
Fix broken YAML from previous suggestions
The 4 suggestions below fix the duplicate/missing lines. Apply all as a batch.
5dfef07 to
6367ca1
Compare
Codecov Report✅ All modified and coverable lines are covered by tests. 📢 Thoughts on this report? Let us know! |
…ase10-workload-validation
…ase10-workload-validation
…ase10-workload-validation
| prod = References(set(), set()) | ||
| tests = References(set(), set()) | ||
| for path in iter_sources(root, REFERENCE_EXTENSIONS): | ||
| text = strip_comments(read_source(path)) |
There was a problem hiding this comment.
String literals not masked in referenced_constants() cause false references in logs. Mask like constant_definitions():
text = strip_comments(read_source(path))
text = STRING_LITERAL.sub(lambda m: " " * len(m.group(0)), text)
if path.name.endswith("SpanNames.h"):| text = strip_comments(read_source(path)) | |
| text = STRING_LITERAL.sub(lambda m: " " * len(m.group(0)), strip_comments(read_source(path))) |
| XRPLD="$2" | ||
| shift 2 | ||
| ;; | ||
| --nodes) |
There was a problem hiding this comment.
NUM_NODES not validated — 0 passes silently, non-numeric crashes script with seq error:
case "$2" in
''|*[!0-9]*|0) die "--nodes requires a positive integer, got: ${2:-empty}" ;;
esac
NUM_NODES="$2"| try: | ||
| custom = json.loads(args.weights) | ||
| weights = {k: int(v) for k, v in custom.items()} | ||
| if not weights or sum(weights.values()) <= 0: |
There was a problem hiding this comment.
Negative weights allowed — random.choices() raises unhandled ValueError. Reject negative weights:
if any(w < 0 for w in weights.values()) or sum(weights.values()) <= 0:
sys.exit(f"--weights must have non-negative values and positive sum, got: {weights}")| if not weights or sum(weights.values()) <= 0: | |
| if not weights or sum(weights.values()) <= 0 or any(v < 0 for v in weights.values()): |
| if (pids != "" && cpu_n != n) exit 1 | ||
| if (cpu_n > 0) printf("%.2f\n", cpu_sum / cpu_n) >> cpu_file | ||
| } | ||
| ' || die "process sampling failed on sample $sample/$SAMPLES: expected $(echo "$PIDS_CSV" | tr -cd , | wc -c)+1 live pids" |
There was a problem hiding this comment.
Arithmetic syntax $(wc -c)+1 appends literal '+1' not result. Wrap in arithmetic expansion:
die "expected $(($(echo \"$pid_list\" | wc -c)+1)) live pids"| ' || die "process sampling failed on sample $sample/$SAMPLES: expected $(echo "$PIDS_CSV" | tr -cd , | wc -c)+1 live pids" | |
| ' || die "process sampling failed on sample $sample/$SAMPLES: expected $(( $(echo "$PIDS_CSV" | tr -cd , | wc -c) + 1 )) live pids" |
…ase10-workload-validation
…e validator The sanity bound required the value to be strictly positive, which rejected three of the four readings the gauge can legitimately produce: a negative count once the validator list has expired, the -1 sentinel for "no published list fetched", and +inf for a config-listed list that never expires. The negative case is the one that matters. It is the signal that expiry has already passed, so the gauge deliberately does not clamp at zero, and a gate that rejects it would fail exactly when an operator most needs the reading. The -1 sentinel already violated the bound and had simply never been hit, because the validation cluster always fetches a published list. The floor is now a century, which still catches a broken clock. Detecting the unsigned wrap this bound used to hide moves to the MetricsRegistry::daysUntil unit tests, which are deterministic and do not need a running cluster.
…om Ok This board's own copy of the Success series selected status_code=STATUS_CODE_OK. Successful rpc.command spans no longer carry Ok: the specification reserves it for an operator asserting verified success and warns a tool may read it as suppressing errors, so instrumentation leaves a successful span Unset. The blob differs from phase-9's, so merging forward would not have corrected it.
… an object Both workload scripts decoded JSON and went straight to .get() or .items(). An array or a scalar decodes fine and then raises AttributeError, which is not a ValueError, so the handler around the weights parsing could not catch it and the operator saw a traceback naming neither the command nor what arrived. Four sites across the two files, all four guarded, and the weights handler now also catches TypeError for a non-numeric weight value. Negative weights are left alone: they do not raise, the type is dropped from the mix silently, and changing that changes accepted input.
…ase10-workload-validation
The harness checked parentage only within a node. Every declared relationship named two spans from one process, so a receive span that lost its cross-node parent still satisfied the contract: it is created either way, with all its normal attributes, and only its parent differs. Declare the three pairs that cross a node boundary, so a send side that stops propagating context fails the run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ase10-workload-validation
| # AttributeError that follows names neither the command nor what | ||
| # arrived. Reject it here so the message carries both. | ||
| if not isinstance(reply, dict): | ||
| raise TypeError( |
There was a problem hiding this comment.
TypeError escapes exception handler, corrupts stats—use AttributeError:
| raise TypeError( | |
| raise AttributeError("expected dict") |
| raise TypeError( | |
| raise AttributeError( |
The $xrpl_work_item variable's description carried a real ticket id as its example. The filter needs no example, so the id is gone and the wording stays.
The $xrpl_work_item variable's description carried a real ticket id as its example. The filter needs no example, so the id is gone and the wording stays.
clang-tidy's readability-math-missing-parentheses flags seven mixed precedence expressions in this file, and warnings are errors in CI. The parentheses match the precedence the compiler already applied, so every static_assert value and the loop bound are unchanged. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ase10-workload-validation
…se8-log-correlation
…ase9-metric-gap-fill
Only SpanGuard carried addEvent(name, attrs), so a call site holding a scoped guard could not record an event attribute. Forwarding overload, with the no-op twin in the telemetry-disabled stub, so a span can be converted between scoped and unscoped without dropping the attributes on its events.
accept.apply was a plain SpanGuard, so it never became the ambient span of doAccept. The spans the function goes on to create inherited the activated accept span instead and came out as accept.apply's siblings, while running inside its own time window. Every guard was scoped before the SpanGuard split, so this restores the hierarchy that design had. Scoped now, so the hierarchy follows the call flow. Drops the parent-context fallback arm with it: the accept context is captured only while the accept span is live, and that span is a child of the round context, so an invalid accept context implies an invalid round context and both arms returned an empty guard.
MonitoredMode::set calls onModeChange on every round start, so the span was created whether or not the mode moved. A node with a steady mode therefore emitted one mode_change per round carrying mode_old == mode_new, which a live sweep confirmed on every round of both instrumented builds. The round span's own consensus_mode attribute is still written on every call, since that is where the round learns the mode it is running in.
… harness The contract's parent field was documentation only, so no check could fail a span for being parented when it should be a root. A live sweep found three spans nested one level off while every harness check passed. The field becomes allowed_parents, a list derived per span from its creation factory and every call path that reaches it, and is asserted per span. A parent on another node and a parent absent from the trace are both inconclusive rather than violations: the receive spans are parented cross-node by design, and a rotation in flight has not exported its root yet. Spans reached by two call paths list every lawful parent. A second gate reads a round as a unit: every required phase child under the round span on the same node, their start times in protocol order, and no mode_change recording mode_old == mode_new. It selects traces that already hold the accept span, since that span always ends after the round span and can land in a later export batch. num_cleared leaves txq.batch_clear's required attributes: the code writes it only after the batch actually clears.
…phase5-docs-deployment
…se8-log-correlation
…ase9-metric-gap-fill
…ase10-workload-validation
… arm went The accept.apply factory no longer falls back to the round context, so the harness contract lists consensus.accept alone. Note rewritten to describe the scoped child as it is now.
| name when it sits outside any namespace and so has no qualified form.""" | ||
| if qualified in refs.pairs: | ||
| return True | ||
| return qualified == name and name in refs.bare |
There was a problem hiding this comment.
Namespace-less dead constants aren't detected: refs.bare includes the definition. Exclude self-references:
| # json.JSONDecodeError - reply body was not valid JSON. | ||
| # AttributeError - reply parsed to something with no .get(), e.g. a | ||
| # JSON array. | ||
| REQUEST_FAILURES: tuple[type[BaseException], ...] = ( |
There was a problem hiding this comment.
TypeError missing from REQUEST_FAILURES when non-dict replies occur—stats not recorded. Add it to the tuple:
High Level Overview of Change
The validation harness for everything above: drives a multi-node validator cluster under synthetic RPC and transaction load, asserts every expected span, attribute and metric actually fires, then gates timing against a committed baseline. Test and CI assets only — no
src/orinclude/changes.No need to review tasklist files.
Context of Change
Up to here, "the telemetry works" rested on reading dashboards by hand. This adds a repeatable check.
docker-compose.workload.yaml,generate-validator-keys.sh,xrpld-validator.cfg.template;rpc_load_generator.py(async WebSocket, configurable rate/mix) andtx_submitter.py(10 transaction types, configurable TPS, periodic sequence refresh), sequenced byworkload_orchestrator.py.validate_telemetry.pychecks the inventories inexpected_spans.json(41 span types, 62 unique attributes, plus parent-child relationships) andexpected_metrics.json(26 metric groups) against live Tempo and Prometheus.capture_timings.pyquerieshistogram_quantileoverspan_duration_milliseconds_bucketandjob_{queued,running}_us_bucket, producing 28 keys: 8 span names × p50/p95/p99, plus 2 job types × queued/running at p95.compare_to_baseline.pydiffs them againstbaselines/baseline-timings.json(populated, captured 2026-06-05) and flags a regression only when both the percentage and absolute bounds are breached.rpc.processis not gated: it is created only on the HTTP JSON-RPC path, which the WebSocket-only generators never reach, so its quantiles were always null. There is norpc_methodsgroup either (FU-4).benchmark.shruns the workload with telemetry off and on..github/workflows/telemetry-validation.yml: build → cluster → load → validate → regression gate → report.Follow-ups
developartifact instead of committing itbenchmark.shproxy measurements with PromQL, and gate RPC method timingsAPI Impact
None — test, CI and documentation assets only.
Test Plan
docker/telemetry/workload/run-full-validation.sh --xrpld .build/xrpld— cluster reaches consensus, workload completes.